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1. Abstract 

Robotic systems will play an increasingly important role In space operations. This paper 
describes the objective and design of a proposed goal-oriented knowledge-based telerobotic system 
for space operations. This design effort encompasses the elements of the system executive and user 
Interface, and the distribution and general structure of the knowledge base, the displays, and the 
task sequencing. 

The objective of the design effort Is to provide an expandable structure for a telerobotic 
system that provides cooperative Interaction between the human operator and computer control. 

The initial phase of the Implementation provides a rule-based, goal-oriented script generator 
to Interface to the existing control inodes of the telerobotic research system In the Intelligent 
Systems Research Lab at NASA Langley Research Center. 

2. Introduction 

The hostile environment of space encourages the use of advanced mechanical and computational means to 
assist man's development of space as a productive environment. The use of teleoperated manipulators, such as 
the Shuttle's Remote Manipulator System, has proven the practicality of this concept, both In space and In 
other hostile environments such as undersea and In nuclear reactor complexes. 

The proposed Space Station represents a completely new approach to space operations. On-orbit assembly 
and servicing operations will expand fn number, duration, and complexity to exceed the capability of manned 
extravehicular activity. The Space Station must have an IntelHoent remote manipulation system to assist 
astronauts in both assembly and maintenance tasks. 

This paper describes the objective and design elements of a proposed goal-oriented telerobotic system for 
space operations. It also describes the Initial Implementation phase of this system in the Intelligent Systems 
Research Lab (ISRL) at NASA Langley Research Center. 

3. Objective 

A remote manipulation system for Space Station operations will be called upon to perform a wide variety of 
sophisticated tasks on a Space Station whose configuration will evolve. A manipulator system must Interact 
with advanced systems and with modules whose form and function are not yet defined. The system should operate 
with minimum human Intervention, at least for routine or prolonged operations. The Space Station remote 

manipulation system must therefore be generalized, flexible, and easily reprogrammable to accommodate the wide 
range of tasks. Very hlghrlevel task-oriented command capability Is required to provide such features. The 
only way to support task-level cotmands is through internal knowledge which supplies the details not provided 
by the operator. The Space Station therefore requires a task or goal -oriented, knowledge-based f -c-mote 
manipulation system. 

Such a system is considerably beyond the state-of-the-art. Since It Is Impossible to plan for all 

contingencies, any attempt to provide near-complete autonomy will result In an extremely cumbersome system 
which will necessarily fall In unexpected situations. A more acceptable concept of a system which cooperates 
closely with a human operator at increasingly sophisticated levels of connand. Oue to technological advances 
in processors, manipulator subsystems, and sensor subsystems, telerobotic systems are now possible in which 
control Is shared between a human ooerator and knowledge-based software systems. Therefore, the objective of 
the design effort is to provide an evolutionary structure for a telerobotic system, i.e., one that can progress 
from strictly teleoperated through phases of serving as an assistant, a colleague, and an expert, to eventually 
serve as a relatively autonomous unit, requiring only minimal supervision. The system should be configured to 
encompass a full range of operational capabilities from basic teleoperation to nearly-autonomous performance 
Involving task planning and status feedback. 
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4. Design Issues 

4.1 Evolution*!? Capability 

A requirement for a goal-oriented telerobotlc system for Space Station is that It be designed to 
accomodate new technology, both In hardware and software engineering, as It Is developed. Such an evolution 
would allow the following roles to be assumed by a telerobotlc system: 

1. Advisor - Upon operator request, provides suggestions for operation sequences and guidance In 
performing tastes. 

2. Monitor * System monitors progress and performance of teleoperator and provides warnings and possibly 
overrides dangerous actions. 

3. Assistant - Operator requests system to perform tedious operations by Initializing state and 
commanding desired actions (from one state to another!. Operator must also be able to reassume control at any 
point or at certain states.. 

4. Planner - System plans and executes operations In response to task requests and with fixed scenarios. 
System appeals to operator when anomalies arise. 

5. Expert planner - Same as planner except that plans are synthesized from generic subtasks and 
alternates are logically derived. 

6. Robust planner - Same as expert planner except that alternates are developed dynamically In response 
to feedback and performance and operator assumes control as last resort. 

7. Learning planner - The system Is able to autonomously Improve Us performance by generalizing 
circumstances of previous tasks. 

The designs of the telerobotlc system must keep In mind the ultimate objectives of the system performance 
in order to build an expandable structure. 

4.2 Shared Control 

A key issue in the design of this system is the concept of "mixed Initiative", l.e., control Is flexibly 
shared between the human operator and the software system. This allows the human to monitor and assist t**e 
software system in task performance, and vice versa. This approach would Increase system reliability and 
flexibility as well as Increasing the credibility of its actions to the human operator. This type of control 
manifests Itself In several ways to an operator, for example: 

1. The software system. In performing an automated task sequence, may request the operator to perform a 
specific sequence of the task that Itself Is Insufficiently sophisticated to perform. 

2 . The human operator can assist the software system In overcoming positional errors due to sensor and 
knowledge base uncertainty. 

3. The human operator can manually override any gross errors by the software system due to catastrophic 
subsystem failure. 

4. The software system can unload from the human operator well-defined but tedious subtasks, such as 
nut/bolt mating. 

5. The software system can monitor the human operator's Inputs and warn of any Imminent operator errors. 

These examples are Intended to Illustrate several different levels of sophistication of the system. The 
means of maintaining both the operator awareness of the current task situation, and the Integrity of the 
software system knowledge base. Is an important research issue. 

4.3 Symmetrica] Structuring 

A cooperative, mixed initiative approach will increase the complexity of the system beyond that of a 

strictly autonomous or a strictly teleopersted system. This is due to the need to share large amounts of 

information between the system and the human operator. This additional complexity impacts the system In 
several different ways, particularly (1) the Initial development of the system, (2) the maintenance and 
expansion of the system, and (3) the use of the system by the operator. 

To manage this additional complexity, the design of the system must be highly modular, both horiiontally 

(l.e., hierarchical modularity) and vertically (l.e., functional modularity). Moreover, it is critical that 

each module be as symmetrical as possible, both. In control and Interface structure, for example, the displays 
for both the task planning and the knowledge bases use the same structure and are accessed and manipulated 
similarly. Likewise, the system executive uses syitmetrical structure in the implementation of user commands 
within various modes. 
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This approach requires less software to be developed and maintained, and permits an easier understanding 
of the total system by the programmers. It also makes the user Interface easier for the operator to learn and 


5. Basic System Structure 

Figure lisa block diagram of a goal-oriented telerobotic system incorporating several types of internal 
knowledge, sensor feedback, and multiple levels of command access. It is divided logically Into hierarchical 
levels which support system evolution and the "mixed initiative" concept. The system of figure 1 operates on 
three conceptual levels: teleoperator, programmed robot, and task executive, which includes planning and 

scheduling operations. At the teleoperator level, detailed spatial motion commands are supplied by the 
operator. At the robot programming level, a robot-level programming language mechanizes more generalized 
actions with respect to objects in the environment; i.e., ACQUIRE BOLT 26. The upper level of the 
goal-oriented telerobotic system, the task executive, is a task planner whicTT uses facts, rules, and assembly 
details to synthesize robot programing language command sequences to accomplish complex tasks. Host human 
operator commands will eventually be at the task level, with the robot programing level used mainly for 
monitoring and debugging. 

Since the goal-oriented telerobotic system Is Intended to be a generalized structure to support operations 
for Space Station assembly and servicing, it is hierarchical and modular. The lower levels may be developed 
first and then used as a foundation for the development of increasingly sophisticated upper levels. The mixed 
Initiative concept permits development to proceed while maintaining system credibility with the human 
operators. 

6. Component Designs 

This section describes the individual subsystems and their internal functions in more detail. The 
emphasis is on the information and techniques used by each subsystem in the performance of the basic task 
execution, regardless of whether the operator or the system is providing active control. 

6.1 System Executive 

The system executive and man-machine Interface are necessary components of the total system. These 
elements are shown in figure 2, which also shows the type of information communicated between the various 
subsystem elements. 

The system executive defines the character of the entire system. Its primary function is to implement the 
mode control for the "mixed initiative" scheme, allowing either teleoperated control or computer control. The 
executive provides the primary user Interface and is thus responsible for command processing and display 
selection. There are two types of commands available to the user within the computer control mode, execution 
commands and interrogation commands. With execution commands, the user controls task performance by the 
telerobotic system. The displays associated with execution commands are concerned with monitoring system 
performance at particular operational levels and in different levels of detail. With interrogation commands, 
the user can request displays of the structure and content of internal knowledge, displays of subsystem states, 
and displays of performance analysis. 

System configuration and resources also vary depending on the control mode. Thus, another function of the 
system executive is to manage resources and resolve conflicts while coordinating prioritized subsystem 
operations and communications. Figure 2 indicates the communication Interfaces and the types of information 
passed between the subsystem elements. Execution commands go to the task planner, the robot programming 
language, and the telerobotic hardware subsystems which return performance monitoring data. The remainder of 
the conmuni cation paths are for the exchange of information necessary for the functioning of subsystems in the 
performance of a task. Most of these are transfers of Information from knowledge bases to the command 
processing subsystems. 

6.2 Knowledge Bases 

The intelligence of the software system behavior is directly proportional to the richness of the knowledge 
base. A primary research consideration is the representational structure and distribution of such a 
potentially complex knowledge base. This design effort explores the use of both a horizontal and a vertical 
distribution of system knowledge. Horizontally, each knowledge base module is distributed in a hierarchically 
abstracted tree structure. Vertically, the knowledge Is divided into three units: (1) a System Knowledge 

Base, dealing with actuator and sensor states gf the robotic devices; (2) a World Model Knowledge Base, dealing 
with objects and various other environmental attributes; and (3) a Task Knowledge Base, dealing with the 
fnteractfon of the system with the environment in performing a specific task. 

System Knowledge Base .- The system knowledge base maintains the sensor/actuator data from the robotic 
system hardware^ This data structure is used as a mailbox for communications between the actual robotic 
devices and the software system. Sensor data from t h e sensor hardware is continually fed into this mailbox, to 
be accessed by the higher-level control processes. Likewise, commanded actuator states from the higher-level 
control processes are placed in the mailbox to be delivered to the actuator hardware. This information 1$ 
globally available, but is used primarily by the robot programming language, which needs position and status, 
and by the operator displays, to communicate system performance. 
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World Model Knowledge Base .- The world model knowledge base contains the system's perception of Its 
operating environment and how to manipulate It. The knowledge must Include: physical details such as 

dimensions. Inertias, and color; locations and orientations; features such as grasp points and holes; 
descriptions of interconnections; and a physical and logical representation of the structure of complex 
objects. Figure 3 shows the structure of the world model knowledge base and lists the definitions of the terms 
used. The actual information, such as Identifier, location and dimensions, it stored within data structures 
describing the members of the various lists. This Information Is supplied by the sensor subsystems and CAD/CAM 
data on the manufacture and assembly of the objects in the environment. The world model Information Is 
supplied to the task planner, the robot programming language, and on request, to the operator interface 
displays. The world model knowledge base Is also the repository for various world states expected to exist at 
specified points of task performance and used to verify proper execution. 

Task Knowledge Base .- In the performance of a task with a telerobotlc system, there will be knowledge that 
deals specifically with the Interaction of the system and the environment for performing the specific task. 
This class of knowledge has been modularized Into the task knowledge base. Figure 4 shows a conceptualization 
of the three knowledge base modules. The task module would tend to deal with elements of the other knowledge 
bases at a higher level of abstraction, and would itself tend to generate more abstracted elements of 
knowledge. Part of this task-specific knowledge must be resident to plan and perform the task, while part will 
be created during the execution of the task. Most of this "created" knowledge will be transient and discarded 
by the system. However, in advanced systems, part of the created knowledge would be absorbed as a more 
permanent portion of the task knowledge base, giving the planner the ability to learn. 

6.3 Programing Language 

TRICCS.- A useful robot programming language should provide the ability to Issue comnands with respect to 
objects In the environment. The language executor would look these up in the knowledge base to translate the 
Instructions to spatial coordinates. The TeleRobotic Integrated Command and Control (TRICCS) language [1] Is 
an object-directed, higher-level command language for telerobotlc systems which Interacts with the knowledge 
base and serves as a target language for the task planner. Tasks would be decomposed into TRICCS command 
sequences, which would be further broker, down into point-to-point spatial commands to the system actuators. 
TRICCS programs could also be written manually, but this would probably not be cost effective except for 
checkout and debugging purposes. TRICCS would Include a subroutine capability to make the language extensible 
by encapsulating frequently used functions. The TRICCS Interpreter would therefore contain extensive debugging 
facilities such as step execution and a conditional execution mode where a simulation rather than the robot 
hardware executes the commands. 

IRSL MENU.- As the ISRL telerobotlc research system has developed, a command language menu for this system 
has evolved. This menu has grown to have numerous commands, each requiring specific submenu sequences of 
varying depths- These menu options Include the ability to select control via any combination of multiple 
sensors and actuators, and to define coordinate transforms pertinent to the environment. 

The robot-level programming language, either TRICCS or the ISRL MENU language, will provide an invaluable 
monitoring and checkout aid. The decomposed task commands will be available to the human operator in a 
readable form. These commands are at a low level of abstraction, i.e., a high level of detail. This code may 
be displayed to the operator prior to execution to check program logical structure. It may also be displayeo 
during execution to provide a trace of the command sequencing. 

6.4 Task Planner 

Task planning must be performed for all stages of evolution (section 4.1). For the advisor, the task 
planner output Is simply displayed to the human operator in a step-by-step form, with the ability to respond to 
clarification queries. The monitor mode requires more aualysis feedback from various sources to determine if 
system and world states and transitions are proceeding according to the task plan. In addition, ft must decide 
if the operator's actions are potentially detrimental, warn the operator of the circumstances, possibly 
override the operator and explain why. 

In the assistant mode, the operator may elect to transfer active control either to or from the system at 
specified points or states. In accepting control from the operator, the system must assess the world model and 
system knowledge bases, comparf-ng them with the specified states to insure that >t knows how to proceed. If 
there is no match, the differences must be portrayed tc the operator and an exchange of information must occur 
in order to resolve the differences. Enough state and task information must be displayed to make sure that the 
operator urderstands the situation. 

The various advanced planner modes (expert, robust, learning) do not Involve significant involvement by the 
human operator. The system appeals to the operator for information or active help only when anomalies arise. 
This should occur less for higher modes where more alternatives are available and are more tailored to the 
situation. The operator will, however, always be able to ask "what are you doing?" and "why are you doing 
that?" types of questions. 

6.5 Displays and Command Interfaces 

The form of the displays Is designed to be symmetric throughout the system and to offer varying degrees of 
detail depending on whether a task Is currently executing or not. Displays of both the knowledge base modules 
and the task sequencing modules are displayed in a tree/lattice form, and commano/feedback communications are 
displayed dynamically in sequenced text strings. 
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The primary objective of the operator Interface Is to transfer large amounts of sometimes complex 
Information In a fast, easily understandable form. It should make maximum use of graphics since humans 
communicate more efficiently In this form; they are capable of absorbing larger amounts of Information more 
quickly through graphic form than from simple text strings. 

The structure of the Internal data Is represented by a hierarchical tree structure (fig. 5) for displaying 
the task planning Information. Tasks and subtasks are successively divided Into smaller and more detailed 
operations. The operator can traverse the tree and examine the sequence of operations at any level of detail. 
A similar tree structure may be developed for the world model Information by substituting objects and parts for 
tasks and subtasks (see fig. 3) to examine the relationship and details of the environment at several levels. 

7 . Implementation 

Given the complexity of the total planner design, and the limited resources for Implementing the design, It 
was decided to use a phased development process. The Initial phase of the development will consist of buildlnq 
a rule-based expert system to generate a script of ISRL MENU components. This script will be available to the 
operator as an advisor on how to perform a specific task. It will also be able to serve as Input directly to 
the telerobotlc system for execution. 

Figure 6 shows the Integration of this proposed planner with the current telerobotlc system 
Implementation. Figure 6a shows the current Implementation: the operator has access to both the telecontrols 

and the menu options to control the system In performing a task. The proposed planner would add a third option 
to the operator (fig. 6b) and would plan task performance based on the capabilities of the other two control 
modes. 

This Initial Implementation will be used to explore several Issues. A primary Issue Is the expandability 
of the system. The ISRL MENU Is continually changing. Options are expanded or condensed to reflect both new 
capability development and the elimination of obsolete commands. Methods will be explored for facilitating 
these changes within the planner. 

Another Issue Is the transfer of control between operator and planner. This proposed implementation will 
have the ability to exchange control with the operator at specific points. This exchange Interface will be 
exercised In various tasks to determine additional desirable capabilities. 

A third issue to be explored by this Implementation is the amount of feedback required within the system 
for task planning. Olrect access to the system knowledge base, a$ well as to the task and world Knowledge 
bases, may be required by the planner. This Is expected to depend on the degree of operator Interaction 
required by the planner. 

8. Summary 

A promising method of developing a robotic system for the Space Station Involves a phased "mixed 
Initiative" technique which provides a gradual transition from teleoperated to nearly-autonomous task 
performance. This approach provides flexibility, redundancy, on-board checkout, and operator credibility. 
This approach also lends Itself to the use of modular and hierarchical structure In the design of all elements 
of the system. Including the knowledge bases, the operator displays, and the task planning. This symmetry of 
design structure can decrease the complexity of the total system, making It easier to develop, maintain, and 
operate. 

The goal-oriented telerobotlc system described Is In the early stages of development. The component and 
manipulator levels ( g. 1) have been Initially Implemented both In simulation [2], and in hardware [3]. A 
robot programming language has been defined and described [1], A world model knowledge base has been designed 
and a report Is In progress. An initial system knowledge base has been implemented and is undergoing 
refinement. 

An initial task planner Implementation Is being developed. This Implementation consists of a rule-based 
export system which generates scripts of menu elements to interface to the existing ISRL telerobotlc system 
Implementation. This initial planner will be used to explore planner expandability, control exchange between 
operator and planner, and feedback /opera tor Interaction requirements. 
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Figure 1 - Goal-oriented telerobotic system 
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Executive, operator, and display interface 
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Figure 6 - Rule-based planner structure 
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